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ADAPTIVE SIDETONE AND ADAPTIVE VOICE 
ACTIVITY DETECT (VAD) THRESHOLD FOR SPEECH 

PROCESSING 

5 BACKGROUND OF THE INVENTION 

1. Field of the Invention 

[0001] The present invention relates generally to speech processing in 
communications systems. More specifically, systems and methods for adaptive sidetone 
and adaptive voice activity detect (VAD) threshold for speech processing are disclosed. 

10 

2. Description of Related Art 

[0002] Modern communication systems greatly rely on digital speech processing in 
order to provide efficient systems. Examples of such communication systems are digital 
telephony trunks, voice mail, voice annotation, answering machines, digital voice over 

15 data links and the like. Such speech processing systems often incorporate a voice activity 
detect (VAD) function, also referred to as a signal classifier. The VAD determines when 
the user is speaking and when the user is silent. The output of the VAD, also known as a 
voicing decision, is binary. The voicing decision may be used to control, for example, 
when to measure the level of background noise, when to suppress sending speech packets 

20 across a wireless medium (silence suppression), when to adapt a speech filter or speech 
beamformer to the user's speech, or when to adapt a noise filter or noise beamformer to 
the background noise. 

[0003] A VAD threshold is used to determine whether speech is present and is a 
critical parameter for the proper operation of these speech processing systems 
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implementing VAD. The VAD threshold may be a single fixed value for all levels of 
noise that is used to compare to a running average of short term integrated energy in the 
input signal over some integration interval, usually a few milliseconds to hundreds of 
milliseconds. The VAD threshold may also be adapted to the noise level as measured 
5 over a long interval, such as ten to hundreds of seconds. More complex solutions use a 
VAD vector of thresholds that is used to compare to short term energy in several audio 
frequency sub-bands and then sum them together in some weighted manner where the 
weights reflect the relative importance of each of the sub-bands. 
[0004] However, one problem with such VAD thresholds is that a fixed value is not 

10 optimal for all levels of ambient noise that may surround the speaker, particularly when 
the noise level is high. Normal speech may include as much as 60% of silence on average 
in a two-way conversation. During the periods of silence, the microphone or other 
speech input device picks up the environment or background noise. The noise 
characteristics and level may vary significantly, for example, from those of a quiet room 

15 to those of a noisy street. If the VAD threshold is too low, then the VAD will suffer a 
high level of false positive errors in a high ambient noise situation. If the threshold is too 
high, then the VAD will report a high level of false negative errors when the speaker is in 
a quiet environment. 

[0005] In addition, in a high noise environment, the speech to noise ratio is so low 
20 that even if the VAD threshold is set to the optimal point, the VAD algorithm suffers 

enough errors that the threshold adaptation often adapts to the speaker's voice or does not 
have a chance to adapt to the unvoiced noise. This tends to draw the threshold away from 
the optimal point, which can further reduce the VAD accuracy. 
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[0006] Thus it would be desirable to provide an improved VAD system with lower 
false positive and negative rates in high noise environments. 



SUMMARY OF THE INVENTION 

5 [0007] Systems and methods for adaptive sidetone and adaptive voice activity detect 
(VAD) threshold for speech processing are disclosed. It should be appreciated that the 
present invention can be implemented in numerous ways, including as a process, an 
apparatus, a system, a device, a method, or a computer readable medium such as a 
computer readable storage medium or a computer network wherein program instructions 
10 are sent over optical or electronic communication lines. Several inventive embodiments 
of the present invention are described below. 

[0008] The voice activity detect method generally includes determining an average 
noise energy level of the communications signals based on noise energy amplitude during 
periods of no voice activity, converting the average noise energy level to sidetone 

15 attenuation, generating an optimized VAD threshold based at least in part on an increase 
in voice level resulting from the sidetone attenuation, and performing VAD based on the 
generated VAD threshold. The average noise energy may be determined by detecting 
periods of no voice activity (or silence), determining noise energy amplitude during the 
periods of no voice activity, and integrating the noise energy amplitude over an 

20 integration period to generate the average noise energy. The average noise energy level 
may be converted to sidetone attenuation according to a predetermined ambient noise to 
sidetone attenuation transfer function and to attack and release time constants. The 
ambient noise to sidetone attenuation transfer function may include low and high ambient 
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noise thresholds and minimum and maximum sidetone attenuation levels such that the 
sidetone attenuation is set to the minimum or maximum sidetone attenuation level when 
the noise energy level is below or above the low or high ambient noise threshold, 
respectively. The sidetone attenuation preferably increases linearly between the 
5 minimum and maximum sidetone attenuation levels as the noise level increases from the 
low to the high ambient noise threshold. 

[0009] The generating of the VAD threshold may include utilizing an estimated, 
actual, and/or an expected increase in voice level based on the sidetone attenuation. The 
VAD threshold may be increased in proportion to the increase in voice level and/or 
10 increased based on the sidetone attenuation. The gain on the communications signals 
may be reduced in proportion to the increase in voice level. 

[0010] The VAD system generally includes an adaptive VAD threshold generator 
configured to generate an optimized VAD threshold based on an increase in voice level 
resulting from sidetone attenuation and a comparator for comparing received signals to 

15 the adaptive VAD threshold to determine existence of voice activity. The sidetone 

attenuation is based on an average ambient noise energy level determined from a noise 
energy amplitude during periods of no voice activity and a comparator for comparing 
received signals to the adaptive VAD threshold to determine existence of voice activity. 
[0011] A communications system incorporating an adaptive VAD system generally 

20 includes a voice activity detector and an adaptive sidetone control in communication with 
the microphone and the voice activity detector, the sidetone control being based on an 
average ambient noise energy level determined from a noise energy amplitude during 
periods of no voice activity as determined by the voice activity detector, the adaptive 
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sidetone control being configured to attenuate the sidetone level with increased ambient 
noise level. 

[0012] These and other features and advantages of the present invention will be 
presented in more detail in the following detailed description and the accompanying 
5 figures which illustrate by way of example the principles of the invention. 

BRIEF DESCRIPTION OF THE DRAWINGS 

[0013] The present invention will be readily understood by the following detailed 
description in conjunction with the accompanying drawings, wherein like reference 
10 numerals designate like structural elements. 

[0014] FIG. 1 is a block diagram illustrating a communications system implementing 
a constant sidetone system. 

[0015] FIG. 2 is a block diagram illustrating a communications system implementing 
an adaptive sidetone system. 
1 5 [0016] FIG. 3 is a block diagram illustrating an ambient noise level detector of the 
communications system of FIG. 2 in more detail. 

[0017] FIG. 4 is a block diagram illustrating the adaptive VAD threshold(s) 
determinated. 

[0018] FIG. 5 is an illustrative graph of an ambient noise to adaptive STMR or 
20 sidetone loss transfer function. 

[0019] FIG. 6 is an illustrative graph of an ambient noise to adaptive STMR or 
sidetone gain transfer function corresponding to the ambient noise to sidetone loss or 
STMR transfer function of FIG. 5. 
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[0020] 



FIG. 7 is a flowchart illustrating a process for adaptive sidetone control 
DESCRIPTION OF SPECIFIC EMBODIMENTS 



[0021] Systems and methods for adaptive sidetone and adaptive voice activity detect 
5 (VAD) threshold for speech processing are disclosed. The following description is 
presented to enable any person skilled in the art to make and use the invention. 
Descriptions of specific embodiments and applications are provided only as examples 
and various modifications will be readily apparent to those skilled in the art. The general 
principles defined herein may be applied to other embodiments and applications without 

10 departing from the spirit and scope of the invention. Thus, the present invention is to be 
accorded the widest scope encompassing numerous alternatives, modifications and 
equivalents consistent with the principles and features disclosed herein. For purpose of 
clarity, details relating to technical material that is known in the technical fields related to 
the invention have not been described in detail so as not to unnecessarily obscure the 

1 5 present invention. 

[0022] VAD thresholds are preferably adaptive and may be determined based on the 
output of an adaptive sidetone control that decreases the sidetone level in a noisy 
environment to encourage the speaker to speak louder and thereby increase the speech to 
ambient noise ratio (SANR). With increased SANR, adaptive VAD thresholds should 

20 improve VAD accuracy and thus improve the performance of speech enhancement 

algorithms employing VAD. In one implementation, the VAD thresholds may be raised 
by the expected average increase in speech level due to reduced sidetone level which may 
be determined algorithmically and/or empirically. However, more complex VAD 
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threshold adaptation algorithms may also be employed although stability problems may 
accompany complex algorithms. 

[0023] For example, a simple linear formula may be used to adjust the VAD 
threshold to an optimal value, as follows: 

5 VADthreshold adapted = VAD t hreshold nominal + A * sidetOne_attenuation 

where A is a constant with a value between 0 and 1 . Similarly, a weighted vector of 
constants of a frequency domain implementation may also be used, as follows: 

VADthreshold adapted(n) = VADthreshold nominai(n) + A(n) * sidetone_attenuation 
where VADthreshold adapted (n) is element n of a vector of adapted VAD thresholds, or more 

10 specifically, VAD sensitivities for each frequency bin of a frequency domain 

representation of the speech waveform. VADthreshold nominai(n) is element n of a vector of 
the nominal VAD thresholds as generated from the background noise estimator. A(n) is 
element n of a vector of constants describing the sensitivity of each frequency component 
of the VAD threshold to the variations in the sidetone attenuation. Those skilled in the 

15 art will recognize that non-linear variations of these formulae may also be employed. 

[0024] Prior to a discussion on the adaptive sidetone control, a background discussion 
on the non-adaptive constant sidetone mechanism is presented with reference to FIG. 1. 
In particular, FIG. 1 is a block diagram illustrating a communications system 100 
implementing a constant sidetone system. The communications system 100 may be a 

20 telephone headset or a handset, for example. Although the communications system 100 
shown is a digital implementation, it is to be understood that analog and/or digital-analog 
hybrid implementations may be employed. As shown, the communications system 100 
includes various analog to digital (A/D) and digital to analog (D/A) converters for 
converting analog signals to digital signals and for converting digital signals to analog 
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signals, respectively. For the signal receiving data path, the system 100 converts received 
signals (Rx) 102 to digital signals with an A/D converter 104. After various digital signal 
processing (DSP), the received signals are converted back to analog signals using a D/A 
converter 106 and output on a speaker 108. For the signal transmission data path, analog 
5 signals received from a microphone 110 may be converted to digital signals using an A/D 
converter 112. The digital transmit signals are utilized for various digital signal 
processing functions and converted back to analog signals using a D/A converter 1 14 and 
transmitted as transmit signals 116. 

[0025] Although the communications system 100 shown is a digital implementation, 
10 i.e., the entire receive and transmit data paths are DSP-based, it is to be understood that 
analog and/or digital-analog hybrid implementations may be employed. The digital 
implementation of the communications system 100 may be adopted for applications such 
as a wireless headset or handset using digital radios. 

[0026] The communications system 100 implementing digital data paths preferably 
15 includes an adaptive sidetone cancellation circuit that includes a sidetone filter adapter 
118 and a sidetone predictive filter 120. The adaptive sidetone cancellation circuit is a 
network line echo canceller that cancels out the normal terminal network induced 
sidetone from the received signals which varies with the receive volume. 
[0027] The telephone sidetone is removed from the received signals because in 
20 communications systems implementing digital data paths, such as digital wireless 
headsets or handsets, the telephone sidetone is delayed sufficiently, typically several 
milliseconds, so as to be an annoyance and/or distraction to the user. In other words, if 
the reflected electric sidetone signal has a noticeable delay, typically 2 or more ms, the 
user generally interprets the signal as an echo rather than sidetone. The term "telephone 
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sidetone" is generally used herein as a combination of what is normally referred to as 
local sidetone coming from the local hybrid and network echo coming from the rest of 
the network. The headset or handset sidetone is the signal from the user's voice that is 
returned to the speaker 108. Sidetone allows the user to hear the user's own voice as the 
5 user speaks. As used herein, "user" generally refers to the near-end person using the 
headset or handset while "far-end listener" refers to the person at the other end of the 
telephone conversation. 

[0028] The communications system 100 implementing digital data paths preferably 
also includes an amplifier 122 with a constant gain 124 to inject a local sidetone after the 

10 residual sidetone from the telephone line is removed from the received signals. In other 
words, a locally synthesized constant amplitude sidetone independent of the receive 
volume control setting is generated from the transmit signal and added to the receive 
path. Because the normal telephone sidetone in a digital packet-based wireless 
headset/handset is often delayed by several milliseconds to be an annoyance and 

15 distraction to the user, as noted above, a locally synthesized sidetone is preferred over the 
normal telephone sidetone. 

[0029] The level of the injected sidetone or the constant gain for the amplifier 122 is 
preferably defined in terms of a sidetone masking rating (STMR) 124. For example, a 
typical STMR for a constant sidetone leveling system is 10 dB for analog telephony only, 
20 12 dB for digital or wireless in Europe, and 16 dB for digital or wireless in North 
America. Because the normal path loss from a person's mouth to the ear is 
approximately 15 dB, an injected sidetone using an STMR of 10 dB for an analog phone 
would be approximately 5 dB louder than what the user hears acoustically from the user's 
mouth. 
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[0030] Having presented a background discussion on the non-adaptive constant 
sidetone mechanism, the adaptive sidetone control will now be discussed with reference 
to FIGS, 2-6. In particular, FIG, 2 is a block diagram illustrating a communications 
system 150 implementing an adaptive sidetone control 160. The adaptive sidetone 
5 control 160 is an ambient noise level detector that measures background noise level 

picked up by the microphone 110. The detected noise level is used to control the gain of 
the sidetone injection. The sidetone adaptation may be implemented in a frequency 
dependent manner in which some parts of the sidetone frequency spectrum are 
compressed more than others, similar to the multi-band compression techniques used for 

10 hearing aids. The background noise detection may also be additionally or alternatively 
implemented using a frequency-weighted analysis. Where both the frequency dependent 
sidetone adaptation techniques and the frequency- weighted background noise detection 
technique are implemented, the sidetone frequency response is preferably matched in 
some manner to the background noise frequency response. 

15 [0031] The adaptive sidetone control 160 continuously varies the amplitude of the 
sidetone level to the headset or handset user by continuously adapting the sidetone 
depending on the level of ambient acoustic noise sources detected by the microphone. If 
the ambient noise level is high, the sidetone level is reduced (i.e., sidetone attenuation is 
increased), which will likely induce the user to speak louder so that the user can hear the 

20 user's own sidetone-generated voice. This adaptive sidetone concept thus encourages the 
user to speak louder in a noisy environment by increasing the STMR which decreases the 
sidetone loudness and thus improves the speech to ambient noise ratio (SANR) perceived 
by the far-end listener. 
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[0032] Preferably, a corresponding reduction in the transmit gain proportional to the 
reduction in the sidetone gain is implemented via a transmit gain reducer 154 so as to 
keep the user's voice amplitude within the normal telephone amplitude range. The 
proportional reduction in the transmit gain thus maintains the speech level presented to 
5 the far-end listener effectively the same regardless of the ambient noise level while 

reducing the ambient noise perceived by the far-end listener. In addition to inducing the 
user to speak louder in a high ambient noise environment, reducing the sidetone level in a 
high ambient noise environment also results in a reduction in the amount of ambient 
noise injected into the user's ear from the sidetone signal, i.e., the user hears less of the 
10 ambient noise through the sidetone, sometimes referred to as listener sidetone. A 

reduction in the amount of ambient noise injected into the user's ear from the sidetone 
signal reduces user discomfort in high ambient noise conditions. 

[0033] By encouraging the user to speak louder in high ambient noise conditions and 
thus increasing the speech to ambient noise ratio (SANR), the VAD threshold adaptation 

15 to find an optimal threshold value is also improved. For example, the optimal threshold 
value may suffer fewer false positives and fewer false negatives as a result of the 
improved SANR. A simpler approach may be to merely raise the VAD threshold value 
by the expected average increase in speech level due to the reduced sidetone level. This 
simpler approach thus avoids a more complex VAD threshold adaptation algorithm and 

20 any accompanying potential stability problems. 

[0034] The adaptive sidetone control 160 is shown in more detail in the block 
diagram of FIG. 3. In particular, the adaptive sidetone control 160 includes a no voice 
activity detector (NVAD) or silence detector 162 which monitors the transmit signal Tx 
from the microphone and detects for silence. When the user is not speaking, the output of 
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the NVAD is TRUE. Note that a VAD function may already exist in the communications 
system and a NVAD would simply be the inverse of a VAD. The adaptive sidetone 
control 160 also includes an energy detector 164 and an integrator 166 that generates an 
estimate of the average noise energy by averaging the noise amplitude detected |Tx| over 
5 an integration period during periods of silence. The noise level output signal is converted 
to the sidetone gain using amplifier 168 according to a predetermined ambient noise to 
STMR transfer function. As noted above, STMR is used to control the sidetone injection 
level. 

[0035] FIG. 4 is a block diagram illustrating the VAD 162 in more detail. As shown, 
10 the VAD 162 includes an adaptive VAD threshold(s) generator 170 and a VAD 

comparator 172. The adaptive VAD threshold(s) generator 170 receives as input the 
STMR from the adaptive sidetone control 160 and/or the expected, estimated, or actual 
average increase in speech level due to reduced sidetone level. The VAD threshold may 
be a single value or may be a vector of thresholds each corresponding to an audio 
15 frequency sub-band. The vector of thresholds may be used to compare to a short term 
energy in several audio frequency sub-bands and summed in a weighted manner to reflect 
the relative importance of each of the audio frequency sub-bands. Based on the output of 
the adaptive VAD threshold(s) determinator 170 as well as the transmit signal Tx, the 
VAD comparator 172 makes the determination whether there is voice activity or silence. 
20 [0036] The expected, estimated, or actual average increase in speech level due to 
reduced sidetone level may be determined empirically and/or algorithmically. However, 
more complex VAD threshold adaptation algorithms utilizing the output from the 
adaptive sidetone control 160 may also be employed. 
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[0037] FIG. 5 is graph of an illustrative adaptive STMR or sidetone loss to ambient 
noise transfer function. As is known, increasing the STMR or sidetone loss decreases the 
sidetone level. Thus, when the ambient noise is at or below a predetermined low 
threshold, e.g., 40 dB SPL, the STMR is set to a constant, e.g., 16 dB. This constant 
5 STMR is typically the same as or similar to what the STMR would be set to in a constant 
sidetone case. As the ambient noise increases from the low threshold (40 dB SPL) to a 
high threshold (75 dB SPL), the STMR increases. In the example shown in FIG. 5, the 
increase in the STMR is preferably linear and the STMR increases to 28 dB when the 
ambient noise increases to the high threshold of 75 dB SPL, thus reducing the sidetone 

10 amplitude with increasing noise levels. Above the high threshold of 75 dB SPL, the 

STMR remains at a maximum constant at 28 dB. As the STMR can be used to control the 
sidetone level, the STMR transfer function of FIG. 5 maybe transformed and described 
in terms of an adaptive sidetone gain transfer function as shown in FIG. 6. 
[0038] It is noted that, for telephony applications, reducing the sidetone gain beyond 

15 12 dB as shown in FIG. 6 in the telephony frequency range may not induce the user to 
speak at a significantly higher level because, at some point, it is too difficult or 
uncomfortable for the user to speak any louder. The maximum level of sidetone 
attenuation or the maximum reduction in the sidetone gain is preferably between 
approximately 10 dB and 12 dB. In addition, at greater than approximately 12 dB of 

20 sidetone attenuation, the user may perceive the user's own acoustic free path "sidetone" 
more than the sidetone from the headset or handset. Thus, further reducing the sidetone 
gain beyond 12 dB for telephony applications may not achieve any further improvements 
in the signal to ambient noise ratio (SANR). 
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[0039] As an example, the maximum achievable STMR before the free acoustic path 
dominates the STMR function may be approximately 28 dB. Thus, the maximum 
increase in voice level achievable by the adaptive sidetone control using a headset or 
handset that has an STMR of 16 dB would be at or around a sidetone attenuation of 12 
5 dB. 

[0040] In addition to implementing the STMR or adaptive sidetone gain transfer 
function, the adaptive sidetone system preferably also implements attack and release time 
constants. The attack time constant refers to the rate that the sidetone gain is reduced in 
response to a sudden increase in background noise and is preferably relatively fast to 

10 ensure that the user begins to speak louder as soon as the increase in ambient noise is 
detected. The release time constant refers to the rate that the sidetone gain recovers after 
the background noise suddenly diminishes and is preferably longer so as to avoid any 
gain pumping effect that may be distracting to the user. In general, the attack response 
may be approximately 1.5 to 2 dB/sec and the release response may be approximately 0.5 

15 to 1 dB/sec. In one example, the attack time constant may be approximately 5 seconds 
while the release time constant may be approximately 15 seconds. In a digital 
implementation of the communications system, a hold-off from attack to release on the 
order of approximately 1 to 5 milliseconds may be implemented. 
[0041] It is to be noted that FIGS. 4 and 5 merely illustrate one example of an 

20 ambient noise to STMR transfer function and a corresponding adaptive sidetone gain 
transfer function, respectively. Different types of headsets and handsets will generally 
require different levels of sidetone adaptation, depending on the particular 
headset/handset design. In addition, various parameters of the adaptive STMR transfer 
function affect the function and performance of the adaptive sidetone control. For 
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example, the parameters of the adaptive STMR transfer function include minimum and 
maximum sidetone amplitude attenuation or STMR and the low and high ambient noise 
level thresholds. Furthermore, the parameters and performance of the ambient noise level 
detector also affects the function and performance of the adaptive sidetone control. 
5 Examples of the parameters for the ambient noise level detector include the attack and 
release time constants, the noise energy integration period, and the VAD accuracy 
including VAD false positive and negative rates. The peak to average ambient noise ratio 
may also affect the performance of the ambient noise level detector and thus the adaptive 
sidetone control. 

10 [0042] FIG. 7 is a flowchart illustrating a process 200 for adaptive sidetone control. 
At step 202, signals are received via the microphone. At step 204, a no voice activity 
detector detects whether there is silence based on the adaptive VAD threshold. If silence 
is not detected, the process returns to step 202 for signal processing of the next signals 
received via the microphone. If silence is detected, the noise energy amplitude |Tx| is 

15 determined at step 206 and integrated over an integration period at step 208 to generate 
an estimate of the average noise energy. At step 210, the noise level output signal is 
converted to the sidetone gain according to a predetermined ambient noise to STMR 
transfer function and according to the attack and release time constants. At step 212, the 
adaptive VAD threshold is generated based on an increase in speech level resulting from 

20 the sidetone attenuation. The process then returns to step 202 for signal processing of the 
next signals received via the microphone. 

[0043] Because the performance of a speech enhancement algorithm is affected by 
the accuracy of the VAD, the accuracy of which is in turn affected by the speech to 
ambient noise ratio (SANR), adaptive sidetone control which improves the S ANR can 
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improve the performance of a speech enhancement algorithm, particularly when the 
headset or handset is used in a high noise environment. For example, any headset or 
handset that uses DSP to enhance the user's speech and incorporates adaptive sidetone 
control, should provide more intelligible speech in a high noise environment. 
5 [0044] While the preferred embodiments of the present invention are described and 
illustrated herein, it will be appreciated that they are merely illustrative and that 
modifications can be made to these embodiments without departing from the spirit and 
scope of the invention. Thus, the invention is intended to be defined only in terms of the 
following claims. 
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